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'ilL DOCUMENTATION ^AND EVALUATION 'OF A SOCIAL CHANGE PROJECT: 



. SrX* EQUITY IN EDUCATIONAL LEADERSHIP ' 

.J" ^ ^ 

^' « 
,.!)> ken Kcrnpner; Rita Poagialcs, and Patricia A. Sclimuck 

Feminist troublemkers h^ye been arouna a lonj, time, ^The work of- Mary 

^Istencraft, Anne Hutchinson, Sojourner Truth, Margaret Sanger, , Susan B. 

Anthony and Emma 'Goldman span the centuries. Although their lives are ' 

known to uS' through accounts of historians and biographers, much of what 

they learned about changing the so^i^ order is lo^st to us. '"Troublemakers 

have be^n ground a long time; however,' evaluators of troublemakers are more 

* <j 

recent. 

. . \ 

Toda> we are 'indeed fortunate. Not only are we paid to be '*trouble- 
ntakers" by our federal government, we arje ^puijred on by federal 4emands - 
for accountability to evaluate oui; efforts to help others be more successful 
^is 'AERA panel on evaluation of pro^t^ams under the V/omen's Educational 
Equity Act (JVEEA) is ain important step in providing a legacy for change. 
Perhaps through evaluations of projects, such as Sex Equity in Educational 
Leadership- (SEEL), yie can help other troublemakers be more^uccessful , 

We offer our remarks primarily to large social action projects. For 
example, funded pfoj^cts under'lVEEA, offer fertile ground for developing 
conn^h and systematic modes of evaluation that can 'be u^ed by others. 
Although each WEEA project deals with* only a sptsVific problem of sex role 
bias (e:g,, rural high^schoVl women or women in administration), solutions 
to. these specific prdblemg may significantly contribute to changing^'the 
re^uni>^rsar and pervasive problems of cjrfrfural sex role bias. -Through 
n- and systematic approaches foo: e^alutating project dealing with. 
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ufcquities in school administration, the penal systenv, the medical profession, 
^and school lifg> vve hope to enable future chang^ a^ents>to solve the uni- 
versal problem of sex role bias.- 

Three characteristic*^ of evaluating social action pro j ects have guided' ^ 
the development of our methodology.^ Firsts evaluation is necessarily 

evaluative. Although that is a tautologidal staten^nt, we think it is a 

\ ' ' . ' • 

fact too often overlooked. The purpose of evaluation is to make judgments 

* \ 

about the role of a program, product, or process. Second, the. purpose o'*f 

^ ^valuation is to provide others with information to make decisions about 

what to do or not to do ^Worthen arid Sande^;^1973) . It is SEEL's intent. 

to judge our failures and successes in Oregoii so that others can make better 

decisions about, how to achieve sex equity in educational administration. 

— ■ — — ™ — , — -----^ — , . ^ 

The third characteristic is the combination of looking at^the., inward processes 

^ *^ ^ ' , 

^ of a project and assessing the .outward impact ^of a project. We refet to the 

description of inward processes as documentatioji and the assessment of the 

outward impact as evaluation. ' ' ' ' v 

' * , ' \ 

• In the following sections we first 'describe the SEj^^^'Project and then 
describe the combination of documentation and fivaluatip?! 4t>iat we are .using. 
We then describe each method in^ detail. Finally,' w§ conclude by sug'gesting 



^that WEEA encourage common and systematic procedures' fbi* documenting an^ 
evaluating large scale social^ction projects. ^ ' * V - ^ . 

■ . .-. > ■ _Y ■ ; 

' ^ \ • The SEEL Project: \k Description/ ' ^ \ 

'The SEEL Project is housed at the Center^ foj Ed^ucatibnal -Policy, and- ' ^ 
Management (CEPM) at the University o'f Ore^goB. » It is funded b^ the 'Women's 



liducational Hciuity Act of the Office of nducation. ^ITL exists to develop 
an'd test strategies for correcting women's notable absence from administra- 
tive positions in Oregon's public schools. The'^tratigies are to'be usablg 



by otb^er stat^ systems, of education. 
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The sex composition in education^ has never been stable. Before the , 

Civil War men were' "school masters;" after the Civil War education became 

. a "woman's profession," Since World War 1*1, men^have again entered the 

field of teaching. In school management,, however, ^the percentage of w6men, * 

.while never large, has decreased dramatically during the last ten years. By 

1974, for example, women held 66% of all positions in education, yet men held 

87% of the princi^palships, and 99,9% of the superin\endencies (Schmuck, "l976-X^^ 

Despite recent efforts to equalize educational opportunities for males 

and females, women continue to l^e underrepresented in school management.- 

Within the last fi\^ years, several studies, have explored the reasons for 

this disparity. One study*, conducted in 1975 by Patricia Schmuck,, identified 

three reasons for the sex imbalance in educational administration in Oregofi: 

' • *, - 

1. Because it is a male sex-typed occupational role,'Komen have 
not aspired to be admirristrators, and men believe it is inappro- 
priate for women to enter those roles. - ju * 

2. Women with certificates in education have less advanced training 
tha*n men with certificates, and when women enter graduate programs 

j-n education they typically do not enter programs in adrhinistration. ' ' " 

3^ There are formal ^ and informal processes of grooming, j*ecruitment . ' 
arid administrative selection at the local school level that perpet-, 
' uate sex inequity and sex segregated jobs. . ^ ' , 

~ < _^ 

The SEEL Project exists in order to correct this sex imbalance. By imple-' 

menting a number of activities that constitute a'^trategy for change, SEEL' 
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is creating a model of effective -products, procedures, and personnel to 
achieve sex equity usable in other educational settings. 
The project is organized around five specific goals: 

1. To build public awareness of sex inequities in educational 
leadership. 

2. To train and recruit women for educational leadership*. . • V 

^3. To build tfhe staters capacity for continuing activities to 
bring about. equity in school management. 2 

4. Txi conduct research about public attitudes toward females in 
administration, selection procedures in administration, and ^ 
women's career patterns. 

5. To disseminate the model through tfrial replication efforts in 
; Illinois, linkages, with other networks, and, a final product. * 

Each year several different activities have teen designed to reach these 

goals. ^ The activities are aimed at a diverse audience of individuals, 

educational groups, an\i institutions. , We assume that lasting- change will ' 

occur only if the relationships among these aifdiences are changed . 

^ Examples of the more than 20 a(?tivitie.s incLude products such as a 

quarterly newsletter, a textbook on sexvlnequities , a slide show, procedures 

such as the Oregon Netvsork ('a communication system to channel information 

about .applicants- and j.ob openi^igs^, and the formation of the Oregon Women 

in Educationll Administration (an organiza^tion of aspiring* and practicing 

women administrators in the state), 

^ The SEEL sta composed oj^five faculty members and five graduate 

students most^ of whom work part time.' IVe gather weekly to share information, 

plan* and make decisions, BeCiaions *are made by consenisus, and responsibility 

for leadership is shared. 



. • ' Oo'cumentation a!id Evaluation: .Beliefs an<t Assumptions , 
-Tli.e documentation and cvaluatioi\ of SECL ia an atteinpt to create a 
'methodology to account for the intricate d^Tiamics, of social change. The- 
' methodology evolved in the SEEL Project after the project was begun. In 
the original' proposal', evaluatio.n followed a traditional performance goal 
model. The evaluation modei?' was too simplistic and,^ fn ere fore ^ insufficient 
te help others replicate SEEL^s efforts. , * 

The model now includes a diig^' pers{)ective'of documenting the inward ' > 
processes and evaluating, the* outward impdfct.^ We believe both are important*' 
in viewing, phange. The. i;nward and oujwar-d * foctis reflects our belief in the 

dynamics of human interaction and responsiveness in the change prcP^ ss, > 

^* ' . • - ' ^ . 

We see' social change as a .humanly created -phenomenon. 'In the documentation . 
^ ' * *. ,***^ 

and waluatibji, a dia3^ec^ ical/vdw of social reality emphasizes the dynamics 

of interaction and responsiveness between the change agehts and the target^ . 

^, ' ♦ * * ' , • 

groups. ^The model 'focuses on the flow between change efforts and responses 

. . > ^ ^ • < . • ^ * : / 

to thpsfe efforts. It is through this interaction 'that SEEL ^developed change- 

str^egies responsive to their target groups. • ' ^ t 



A major belief of the SEEI; Projept and the documentation, is* the ideology 
of feminism. We presume sex equity' is a s*oci4r necessity , Although we are 
aware that alT WEEA projects operate in a social tohtext of sex role bias, . ^ 

also operate xinder the^ assumption that we can^attdck the specific issues ^ 
of sex -role bias within education. * ' V ' ^ ^ ' 

^ - S"ex bias can be considered as^a spec^ific phenomenon present in some . 
aspetts'of our culture (i.e., job discrimination in education or differential 



treatment in classes), or as a uni ve rsa 1 phenomenon representing inequality 
in all spheres of th,e society. As a universal phenomenon, sex bias per- - . 
petuates women's inequality. Adopting such/ a universal explanation would 
lead to strategie^s .that are not confined to t"he profession of education. 
Our major ass.umption is that if we can begin to 'solve sex inequities within , 
a' specific sphere of^the society (i.e., school -management) we -may >b eg in to 
effect k change in universal problems of sex rolei> bias ^in the more pervasive 

/cultura. ' ' « 

In the next section we intr'oduce the concept of social change as a 
dialectic*. Befcausc the strategies of changing woineT)'s- representatipn in * 
school management are reassessed from contact with SEEL's target groups, 
the documentation and evaluation must account for*this dialectic. Change 

, is v^iewed as evolutionary^ a gradual process of mutual adjustment. 



Social 'Chan|e as a Dialectic 

1^ 



Social interventions are interactions between agfents of, change and 

target group^s. This^dialectic is il-'-us't rated in* Diagram 1 which (1) out- 

lines the initiation- i^sponse process of an intervention,' and (2) identifies 

the inward and outward 'levels of an interaction* The diagram is important 

for documeiytatijjp and evaluation because it accounts for the inward processes 

of strategy de'^elopment and' the outward impact of these strategies on the 

• * * 

target group. 



^ 'Diag^a^ 1 
Dialectic of Social Change 
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jiiitiation-Res^nsc Process of Social Interventions ' 

The' initiation process mf a social change effort is illustrated in 
the right, upper half of Piagram 1. Because change agents, such as SEEL,^ 
^hold certain beliefs and attitudes concerning an ideal social system (i.e., 
women should be eqiia^l to men), internal policies, laws or behaviors. wi 
be^^^rmulated to alter a social .system" that vio_Mtes the ideal. 

The response of ^the target group to the^e change agent 's* behavior is 
illustirated in the louer, right half of Diagram 1. Because the target 
g:foup menjbers hoj^certain beliefs and attitudes (i.e., women cannot he >^ 
high school principals because they cannot discipline boys), the impact 
of ttie interv^ention "will cause some reaction by the target group sxich as 
violence, apathy, disbelief or verbal abuse. Indeed, the r'eactibns of dndi- 
viduals faced 'With s.tress (i.e., disbelief shock, anger)' are not atypical 
of a group's reactions to data-wliich are contrary to their beliefs . For 
insta^nce, for many male 'administrators, atypical rationalization is that 
* "women -don't wafit to be superintendents*" Internalized reactions (fiom. 

• women educators to. information about sex" inequities- in administration are 
^ often different and' lead to Responses' such as, '^KTiat can we ^ change 
I the situation?" ^ . - 

^ . The initiation of the target group's response to the i^tervei\tion 
emanates from their beliefs and attitudes leading to certain policies and' 

^ • • V / _ * . ^ 

behaviors as illustrated^in the lower left portion of Diagram 1. Based 
upon their ass^ments of an ideal social system, target groups formulate 
policies in "reaction to the intervention' efforts . 
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. The dialectic nalure this prbcess is 'apparent \vhcn t'he charrge' agents 
react to the target group^s behavior. For example, SE15L created a'directo:^^ " 
of qualified women administrators in response to the mal,e administr^t^or's con- 
tention, "We nduld hire women; but qualified women don't apply." This 
contlnuoDs dialectical exchange between the twa grbups. can proceed until 
compromise ^ is .reached, one side .domina^tes the other, or one side concedes. 
TTie process may never stop. ^ » " . *V 

Only one target group has been illustrated in^'the diagram to simplify 
the description of the interactive process between the change' agents and 
.the target^group. In reality, however, different grpup^;, comprise the , target, 
audience. By spli'ttin^g the model' below ^thjs beliefs 'and attitudes of the 
. change agents, multiple target groups can be accounted for i;i an intervejition. 
ChangQ agents .vfill develop differing strategies to deal with each target 
♦^oup and i^eact to these groyps by .reformtilating new policies '^d-^behaviors..^ 

Levels of an Intervention ■ v . ' , ' > 

The second purpose of the soci-al change dialectic' enables evaluators* ' 
■ to identify the inward processes" and tHe outward -impact of an intervention. 
The inward processes include beliefs,^ attitudes, and policy formulation/ 
internalization that occur within the change/group and target gro.tq). Inward 
processes are relatively intangible, and primarily' qualitative .methods are ' 
us6d to document this realfn which is identified .in Diagram ^. 

The projectjs impact is more 'observable since phenbmena such as polij:y 
implementation/reaction a'nd behavior can be quantified. Outward evaluation^ 
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can be accomplished by accumulating more objective data on each group's 
behavior and is identified in the middle, area of Diagram 2. 

. By docume;iting the inward processes and evaluating the outward impact ' 

of each group in a social intervention, the more intangible processes, suchv 

* " ' ^ 

as beliefs, attitudes^ policy 'formulation/intemalization as well as the 

■ ^ 

ob§ervable behaviors and reactions, can amplify a'^ de script i6n of the change 
effort's effectiveness. . ' ^ . ' 



Documentation and Evaluation Methodology: ' 

The Use of Qualitative and Quantitative Methods * " 

Social interventions are typically complex treatments of systems, nojt 

experimental treatments of individuals. 'Riecken (IsV^^i^suggests a.tripartate 

distinction;of goals to match the complexity" in changing social systems. 
♦ 

Social intervention goals can 'be classi^fied as 'Hiltimate goals, ".'"instrumental 
goals,'' and "performance goals^" .At the first level (performance goals), daily 
activities are accom^ished or performed to implement the second level ' (instru- 
mental goals). The instrumentalj-goals in turn provide the "principle means" 
Xo accomplish -the third level, ultimate goals. Of^n- onLy performance goals 
are' Evaluated in an intervention and success is determined on how well indi- 
viduals accomplish these daily tasks or goals.. Complete project evaluations 
must^ include, however, an analysis of the^manner in which the accomolishment 
of the performance goals assisted in completing the instrumental goals of ' 
a project, yand . finally how the instrumental goals assisted in completing tl^e 
ultimate goals. For example, one performance goal for SEEL is publishing 
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a quarterly newsletter. Tliis activity is performed to assist the instru- 

mental goal of criming greater ^awarenesj of equity issues in education. 

Thi<s instrumental goal of awareness h^s been developed^to assist in the 
« « # 

» » i * • • 

accomplishment of the ultimate go^l--sex equity. / 

s, ^ - , ^ :^ 

^ These go^als can be evaluated best by combining quantitative and quali- 
tative met}«)ds. Often only quantitative measures are used to assess goal ' 
accomplishment. Qualitative methods are. necessary because purely quantita- 
tive measures are often inadequate to portray social reality. Because the . 
^ility of reactive measures to predict individual l?ehavior in a natural 
environment 'is que^^ti enable, the evaluation of an intervention cannot be 
based only upon aftitydil^al or reactive quantitative mfeasures. Because - 
charisma, coil?)etency, organizational dynamics and social' norms influence the, 
success of a project*.; the effects of these intangible attributes must be' ' 
•dofcumented. Evaluafors that only use reactive measures and ignore the intan- 
gible processes' of the intervention, the^oups. and the environment will 
overlook the complexities that interact to cause behavior • 

6y combining both quantitative and qualitative methods in documentation 
and evaluktion, intangible attributes, internal processes, and concrete 
behaVior tan^e assessed to" evaluate.the impact on, the ultimate, instrumental 
and pierformance goals of a »fi^cial intervention. -The. complementary nature 
of these methods are described by Rist (1977): 

It may well be that som^e of the most intellectually stimulating 
and exciting developments in educational researcjt) over the next 
decade will be* in work-ing out the dmplicatiops of the dialectic 
••[between <iualitative and quantitative methodologies] . If break- 
throughs are to come ,^ they will happen, as Ku,h5^(1970 :110) suggests, a, 
wljen 'scientists see new .and different things when looking with .. . 

r 

- 14 
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' familiar instruments in pl^acjes they have looked before. • It ^ay 
. well be that when Xhe * familiar instruments' of quantitative and 
<^ qualitative methodologies are juxtaposed, we will ',see ^new and 
different things'.' (p. 48) . • , * 

The following section further explains the rationale for using both quali- 
tative and quantitative procedures in documenting the inward processes of 
a change effort and evaluating thd* outward impact of a project. ^ . 
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Inward Process Documentation 
^ 



The inward documentation of g social change project describes the" 
' process and development of policy. Our view that social change is created 
through the dialectic ;0f hunlan 'inter^tions is central^ to the inward docu- 
mentation. This view helps explain the natul^e of staff processes, which 
is both interactive and evolutionary. V/e believer attention to the inward 
subjective life, ideology and history, are critical in understanding how^ 
a project's dynamics Jead to poli^cy implementation. Policies evolve because 
people^ in fl^ence each ot^r's beliefs through interaction amoTTfe individuals, 
within the group and between the staff and tar|;et groups. 



Document action by' Ethnography> 



Ethnography is the method of qualitative research used to document the 
internal reality of *SEEL, The methods commonly used by ethnograj^ers 
include participant observation, key-informan^ts, life history, formal docu- 
ments, unobtrusive "measures and artifacts (Pelto, 1970). The research^oal 
is to portray the full, actual, lived reality of a social situation: ' 

The term ethnography belongs to anthropology: ethnography 
provides the basic descriptive -data on which cultural anthro- 
pology is founded. And ethjio-graphy is, literally, an anthro- 
pologist's 'picture', of the way of life of some interacting ^ 
human group; or viewed as process, ethnography is the 'science 
'of cultural description, ^ 

--Wolcott, 1^75' ^ . 
The SEEL documentation consists of observations, ^nterviews, analysi$ of 
written records and tapes, and instrument^guch as sociograms. 



/ ^ There are tvyo stig'es to the-method: data collection and interpretation. 
In the first stage extensive observations and data are collected and gate- 
'gorizeH..^ Ilii-^^tage' represents^ the traditional boundaries of ethnogi;aphy , ' 
For the SEEt documentation,- a)i interpre,/ativ^ level is' necessary to allow 
the documenter to draw from the observable world to analyse the subjective 
variables involved in^ social change. , - \ 

^ In the SEEL documentation the interpersonal and depision making pro- 
cesses which underly. activities are carefully observed, yet not all 16 
activities of the project are documented fultj)^. Those activities that are 
critical An reaching the ultimate goals of the project, or microcosms of 
project goals,' were chosen. ^ Some of the more complete documented activi- 
ties include the Oregon Network, the formation of Oregon Women 'in Educational 
Administration^ (OWEA)^ th^ teaching of a university class, and the development 

of a slide show. 

. ^ I .V 

Documen'tii\g^th'^fee activities and 'the underlying processes require a 

thorough historical and descriptive^^' record. For iSistance, th6 Ore^n Netftork 
9nd the organization 'of "OV/EA were documented through recorded observations, ' 
tapes and written documents which contained the. Setails of how these organi- 
zations, developed. - Within the I'field notes" are^ the; statements about the, ' 
problems, strategics and* beliefs -af women administrators as weliN as bbser- 
vations on the setting, the dress, non-verbal behaviors, jokes, language 
and speech .patterns . The field notes are messy and disorderly, as is the 
world they reflect. Rather t"lian prematurely impo'sing -order on the internal 
realH-firof SEEL, the documentor attempts to reflect the project's rea'iity. 



As in most life experiences, the initial ciisorder' clears as consistent ' * 
patterns of behavior emerged The process of distilling the dbservationar * 
.data has'**led to more generalizable sta.tements' about SEEL. The potpourri 
of unique happenings becomes recognizable patterns of fcehavior. As the^ 
documentation continues and more observations can be madj, the reality of 
the project becomes more concrete. ' — 

Documentation can assist others in replicating some SEEL activities 
(the ultimate go^l of the project). The requirements of enorgy, planning 
and designing an operation as complex as ihe Oregon.^etwork, for example, 
must be communicated to others. Far some of the activities, such^s the 
development of the newsletter or the slide show, the documentation will be 
primarily descriptive. For other activities, such as the development " of 

' the Oregon Network, ^interpretation will be added to the 'observational 
data. , y/hile'^the mechanics of oi^anizing and implemeTiting , an* activity ' 

\as complex as. the Oregon Network are important to describe in the documen- 
tation, the subjective factors must also be known to fully understand the* 
organization. A descriptive approach alone will not adequately prepare 
others who^ attempt to replicate activities within thQ*ir.pOUTi unique settings 
because each procedure is established within a specific political ^d. 
social context.. 
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Role of the\ Documenter 



A critical issue in ethnographic research is the influence of the docu- 
menter; ethnographers ^use their own perceptions as the criteria for gatherin 

V 

and interpreting the "data, '"n is a subjective method without the rigor- of 
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the scientific method. "Tliis repx£is<?nts both a strengtli and weakness of 
ethnography. The Strength lies in the subjective description; Qjhnographers 
capture not only the obvious' but .also the subtle characteristics of human > 
beings within their environnent. The strength of this method is also its 
weakness. Documentefs bring to ^th^i^ a prefigured notion of 

reality; one's awareness is a built-in bias. * 

This presents certain conflicts and issues. By tradition, ' an ethno- 
grapher maintains a distance from the observed group to guarantee a minimum 
of emotional involvement. It isj^^^^jomed that distance provides a more 
objective and thorough description. The conflict is that a degree of involve 
nient is 'essential to meaningfully experience^ a" group (Geertz, 1973). ^Vhile 
neither extreme is helpful, a Inovement between involvement 'and distance is 
ideal because it allows for an objective perspective as well as a sufficient 
immersion in the realvfey of the project. The dichotomy between "stranger 
and friend'* (Powderm'aker, ^1966) is an essential working style of ethnography. 
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* , , ^ /' * Outward Impact Evaluation " ' <^ 

The out^^d imiaact of an intervention can be e^va-luated in *a b.ehavioral 
and a conparative^ asseissinejit of the change effort.' The behavioral assess- 
ment includes evaluation of th'^resent behavior of both th& change agent's 
and target' groins and the future, potential for change in these group'-s 
behaviors. Tlie comparative ^assessWnt contrasts the chanige effort with 
other possible methods of inteiTventionX ' 

Behavioral Assessment ' ' ' \ 

'Because behavior occurs throl!ghou^p^4dme, an ev^li^tion of an inter- 
vention must include an . assessment c^^4d[>esent -behavior- and, a pre"3ictiori o'f 
future b^havio.r. ^ ' ^ . ' » "-^ - y . , 

Piresent Behavior T Th^ impact &E a. project's present ' intervention . ^ 
activities is observable and quantifiable. Sinde reactive or projective 
measures of behavior or attitudes provide doubtful surveys of reality, 'un- • 
obtrusive measured offer a better method for evaluating behavior. Unobtrusive 
measures are preferable because they do not rely on t he ac^v^cy of predic- 
tions and are not confounded by^ndividual perceptions or acquiescence to 
evaluators . ' * ""^^ — / ' ' ' ' ^ 

When social interventions have well-defined ultimate, instrumental, 
and performance goal^, a project's effectiveness is more easily assessed. ^ 
B/ identifying the cause-effect relationsTlip of change efforts the evaluators' 
Can assess the. impact of project activities. The identification of .goals ''^ \ 
and acti,vitie.s enables the evaluator to measure" the impact of these" strategies, 

■ ' 1 
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* * 

In the SEEL Project, five objectives accompanie^d 'by specific activities 
provide measures for evaluating the present impact of the prbject. Evaluation 
-is accomplished by measuring the" impact of ^the^pe^foirraance gbals (i,e,/SEEL 
activities) and the i^fla^:t of the instromental goa^s^(i.^,^ SEEL objectives), 
^ ^ Examples of. the present behavioral a^.sessment of SEEL^s- activities v 
^ include, the gathering of da^a dn enrollment in certification,,programs, hiring 
practices of school districts, and the plumber of women applicants for admin- 
istrative programs. With counts of product j^e frequency, feedback, requests 
"rfbr information or 'assistance, publication and research usagis^lnd other - 
measures, -the present behavioral impact can be assessed^ 

Future Potential for Change .. An' a^s^ssment of the present change 
behaviors fails to account for the potential, longer range effects created . 
by an intervention. For an intervention to be l^ccessful , it must create 
^ change that endures. The evJf^tion of an intervention's present accomplish- 
ments without the assessment of the potential for continued change would 
provide ai myopic evaluation. Documenting this legacy -for future- change , ^there- 

fore, is a vital component in evaluatio^, * 

* ' r- . ' • # ' 

% , A project^s processes , ^ products , and people provide the eyaluation 

measures for future change, Becaus^ most externally* fun|^ed projects are 
'.ephemeral social Change efforts,^he most effective projects will be those 
^ that leave a legacy of continued change. The ^va^luation of this change 

legacy can b*e accomplished by assessing the effects of the processes, pro- 

ducts, and people of a project to assure future qhange-. 
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An example of a process tKat may facilitate continual change is the' 

■ ' / ^ ^ 

^•Hawthorne effect," found in a clasjsic study!;by RoetWinsberger and 
Dickson (1939). Tliis study illustrated the confounding -effect of the' 
experimenter's_presence.. V/hile it is a problem^'for^rigojo^s "^research " 
designs', such an effect can be desirable in social acti-on. praj ects , For 
example, should administrators feel obligated^to hire women because of a 
watchful SEEL Proj ect, their increased hiring .Of womeh would be a d^irable , 
outcbme. , Sflould adnSnistrators return to their' sexist h^its after the^^ 

SEE-L Project ends, however, the intervention will have had little*residuar\ 

^ . ' ' * • ' ^ . ■ y 

effect. If the project creates methods to continue, th%*' intervention process, 

an organization other than SEEL c6aid. provicle the -"watchful' effect , 'Samples 

of SHELLS efforts. to maintain a/'watchful eye" after funding ends include 

changes in certification requirements , continued publication^of a newsletter. 

creation of a network to monitor adminis^l^ative openings, and the organiza- 

y > _ • ' J ' ' 

tion of a new state^wide admiyistVative wome^n^s^'br^anization, ^ 

In addition to processes that can *kssfire the continuation of kn inter- 

vention, products a^i^st in^proyiding long-term chaVgQ^ 'Ejc^ples of enduring 



products that facilitate contiflilk^l change inclTLide'^the following:, training 
aids, research, publications, textbooks^ slide, show,|p movies, sHort documen- 
taries, or .other such products. For SEEL, '-^th^e^f^nSuring products^can 
cpntinue to raise the public awareness oi^|:hfe* inequii:able' situation and 
supply materials for future change agents ^-^to continual ly. pressure target groups 

Closely associated with process ' and product | ^e people, 'because pro- 
cesses and products are of n6 value' ^unless used, the cultivation of 
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/individuals to continue the legacy of c^iang? is .imperative. 'Th6 establishment 
of a future co^rps 6:t change ageifti^is es'^ential to assure a proiect's 



-legacy. By training individuals for specific positions or organizing groups 
to continue change activities, the int^vention process can continue. SEEL, 
^ for example, has developed the capacity of many individuals and groups to 




continue the legacy of C^ang^u^ Jpij^e Oregon Network has beert developed, sped- 
fically, to create a* pool of '£iualified women^to enter administration an^ 
raise the awareness of present educlational administrators and college .professo: 

By providing processes, products afid^ people tcr continue the interven- 
tion effort, change agents , can better ' assure the future potential fot change. 
In evaluating a change effox\, therefore, the assessment oi>f)rocesses that 
will assure future and continued change must be measured by ^he evaluator* 
By understanding how change occurs the. ^valuator can determine .what processes'' 
are present in a project that will facilitate future change. 

Comparative Assessment ^ • 

• — J ■ ■ -. - 

In addition to a J)ehavioral assessment ot the outward impact, a second 
component comparing possible intervention strategies is necessary. By con- ' 
trastin^altemative intervention-methods, the comparati^ effectiveness 

of a change^effort can -be determined. 
> f ' " , 

If a, social intervention. could be acconplished in a controlled exper- # 
imental setting; fewer problems- would occur in choosing the most effective 
treatment. Without an experimental design and^ the lack of multiple treat- 
ments from wh^^S^to thbose, the most effective' intervention stratej^is / 
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liifficult to clctcrninc. All cxampte of an experimental design cauld- include 
- . I ^ , . / . . 

I ' contrasting the use of federal fui\ds to place women directly into admin- * 

i3trative positions In one state compared to a SEEL Projebt in another state 

which' would p^rovide useful information. The state withNJthe highest number 

of women administrators at some specified time' could be considered th^ most 

^ • . ' ' - . 

successful . ' , ' ' • ^ 

Although such experimental designs are costl^Hiand often not feasible 

* , for social action projects, comparisons ^c an be made between strategies in 

two phases. The first phase is ,a comparison of the performance goals' ^d 

-the instrumental goals within a project: %y determining which activities , 

are most effective in reaching the ultimatd goal, a comparative evaluation 

• of all the project's efforts can be accomplished. 

1* * ' ♦ - 

In the second phase ^<^f comparison, the effectiveness of a pi^ject's 

performance and instrumental goals can be compared to similar intervention 

t* , " ^ ^ " 

^ efforts in other locations. The previous example in v^rhich a SEEL Project 

in one state was compared to the placement of women in administrative posi- 
^tions in another state is^^ example of this second type of comparison, 
tfhobtrusive measures can be helpful in comparing strategies between groups 
by c'oittrasting, for instance, the historical data between states. In the 
SEEL evaluation, unobtrusive measures from state department records are 
being used to, compare the number of 'women hired for , administrative positigps 
between similar states (e.^., Oregon and .Wa$hingt^on) . By matching historical 
data and intervention activities and by comparing the impact of intervention 
strategies used, the most effective strategies can** be. determined. IVhen no 

•A ■ . - 
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similar strategies bctKcen groups exist, nssessnents must be made 'by evalu- 
-ators to .determine whicih strategies provide the preferable- impact , ^ 



Final Evaluation Questions 
Finally, twcr ad'ditional questions must be raised in the evaluai;ii)n 
of social action projects: (1) what is the generalizabdlity of the findings, 
and (2) what is the social efficasy of the project? . 



Generalizability and Adaptability • ^ j ^ 

Although some interventions may be restricted to isolated, social situ- 
ations, the process and products developed by a piioject should be of value 
to people in other locations. No ;single intervention strategy can be effec- 
ti-ve for all conditions and .locations, yet is is important to consider the 
, appli^ility of the products and procQ$se^,,deve loped. Since SEEL ^has been 
funded to di3seminate an intervention modpl, the usefulness and generaliza- 
bilit/ of the mod(?l for change agetits beyond Oregon and the Northwest states 
must be considered. Change agents in different environments with different 
resources must also be able to .adapt th^ SEEL-model and expect positive 
results. 



Social Efficacy * , 

An evaluation of the s^ocial efficacy of a project, should, be included 
in assessing the accomplishment of ah interventioir^'g'ultimate goM. This 
"so what" question concerns the worth to society;of a project's efforts. 



• In this totSl evaluatjcfn ofv a •project, the efficient and the' sufficient 
use of rpsources should be determjjied. . ' , ^ ^ 

In the evaluation of ^ project's efficacy, the social cost involypd 
. must be compared to th^' benefits' derived: was the intervention an efficient 
^ use of resource a- -people;^ time^ facilities, money? With regard^to SEEL, 

the evalu'lition should determine if SEEL has been an effective .use of federal 
ccomp lining sex' equity. , ' / V *' 

An intervention 'ffiuit also be assessed according to its S^ufficient use' . 

' ' ^ -^s >- - . ^ • • " ^ 

of resources: did the project adequately -use all resources available?- To 
effectively accomplish a social intervention, change, agents must be able to 
identify human and ma.terial fesourci^^^han can assist in the change effort, ' 
SEEL, for exampje, has consulted with sch/5ol administrators regarding devel- 
opment. of the best intervention methods. Contact also Ji^s been made \-^it\x 
^ future administrators enrolled as students or employed in lower-l^el, up- 
wardly mobile '^idministirative positions. By determining which individuals 
- and materials can assist ^ffectivej^ip intervention efforts, evaluators 

<^ •-' ^ ~ : ^ ' . 

^ can as%pss the use of resouixes by change agents. . ^ , ' ' ^ 



efit, t^i 



:jfusric 



Conclus-iop — 

> . 0ur federal govemmefit, ^rfcugh IVEEA^ has taken an active role in 



.supporting *^^it>ublemaker*» ppjects attempting to create s. ex role changes 
. in society. We are pleis^^^^be part of that ej^ort. 'Yet, we are also ^ 
*»>C^oncerr)^d aboiTt providing judgments regarding ''the worth of pro^-ecW that 
can help others. make decisidfcs^f or ^continuing change efforts. We suggest 
common^ and systematic evaluation pr6cedux:e would* be hejpful. # 
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In this paper, we have argued that documenting the inward processes 

- • t 

of a project is as important as providing evidence of outward impact using 
both qualitative and quantitative methods, hav,e pres-ented a model of 
the social dialectic to account for policy development as an" evolving, 
mutually adapting system of change, and described in detail the methods of 
inward documentation and outward evaluation. * [ 

J Perhaps i,f all large social action projects had comparable evaluation 

designs, comparisons could be mad^ about which strategies are most effective, 
For instance, there are at least four U'EEA projects about sex inequities in 
, educational leadership, offering different solutions in different locations. 

If comparable data about the inward processes and evidence of outward impact 
could be gathered from each project, others may be better able to judge and 
make decisions for future strategies of change. Thorough evaluation proce- 
^ dures' m'&y provide, our legacy for change. 
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